#407 added option to reduce flushing of streams during write operation. #432
Conversation
src/Parquet/ParquetOptions.cs
Outdated
| /// <summary> | ||
| /// Reduce flushing of stream during write, to increase performance of for instance Azure blob | ||
| /// </summary> | ||
| public bool ReduceStreamFlushing { get; set; } |
There was a problem hiding this comment.
I think it might be better to reduce flushing by default and have no option - less permutations to support.
There was a problem hiding this comment.
i took the liberty of doing so, hope you don't mind
There was a problem hiding this comment.
Sorry but I dont agree, since it was found out that removing that flush will make parquet.net hang in this method call
public Task WriteColumnAsync(DataColumn column, CancellationToken cancellationToken = default)
in some schenarios. So I still believe the best solution for now is to have the flush handling configurable, until a permanent fix for the stuck Task has been found.
There was a problem hiding this comment.
Sorry missed this issue. I'll add it back now and release a patch. Is there any way we can reproduce the failure with a unit test btw?
There was a problem hiding this comment.
Please have a look at the code I sent in a mail to you.
No description provided.